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ABSTRACT 
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technical problem-solving skills through intentional transfer of 
existing knowledge and skill to novel tasks. Sherlock 2, the model 
devised, consisted of high quality device models using illustrative 
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tutoring the trainees. Coaching on how it works was presented 
periodically throughout the tutoring sessions. Reflective follow-up 
included reasoning behind the expert solution and general 
troubleshooting principles. Partial results of an evaluation of the 
intelligent tutoring system showed that the tutor's learning 
environment is concordant with views from the acquired skill through 
coached apprenticeship activities. Instructional content consists of 
authentic problem-solving scenarios that situate trainees in 
realistic contexts where they can practice and hone complex 
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is general izable to a novel equipment system. Similar 
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TUTORING FOR TRANSFER OF TECHNICAL COMPETENCE 

COACHED APPRENTICESHIP AS A FORM OF CONSTRUCTIVIST TRAINING 

The constructivist movement has grown largely as a reaction to problems 
in education that come from excessive emphasis on algorithmic performance and 
rote memorization as the core content of education. It has been noted that 
saying the words of a principle does not produce understanding of what those 
words mean. Also, as long ago as Whitehead, it was observed that having 
learned something in school did not imply that one would use the acquired 
knowledge when it was relevant in real life. This fundamental problem, that 
schooling seemed too internally focused and did not prepare people for life 
outside the context of school performances, led to the constructivist 
viewpoint that students must construct their own knowledge and that education 
consists of providing appropriate learning situations that afford a student 
opportunities to develop personal knowledge that will be useful in later life. 

At one extreme, the constructivist approach takes on an idealized 
humanistic character. Within this view, it is thought inappropriate for the 
teacher even to have explicit goals for learning. The student is to be free 
to develop his own mind, his own understanding, his own competence. The 
viewpoint we have taken is perhaps closer to the other extreme. We realize 
the need for the student to build knowledge anchored in his/her own prior 
knowledge and understanding, but we see many situations in which a culture has 
specific learning goals for its novitiate and in which seeking instruction 
into that culture implies acceptance of at least a partial goal structure for 
learning. 

Even in this more restrictive view, a constructivist approach is 
powerful and perhaps necessary. The knowledge a student will acquire must 
still be anchored in his/her experiences. The nouns and verbs in statements 
of principle must still have meaning for the student. At the same time, the 
learning environment must afford opportunities to reflect on how a particular 
community of practice talks about the world, how it represents the world, and 
how it determines how to act in the world. We have focused our attention on a 
particular communiiy of practice, technicians who use and maintain specialized 
electronic equipment that itself is used to facilitate testing and repair of 
aircraft navigation equipment. Because this community is defined partly by 
its responsibilties to a larger community that includes pilots and 
tacticians who count on having working aircraft, effective practice is 
constrained partly by cultural responsibilties. It is also constrained by the 
need t^ be able to talk to other conimunity members about the work of the 
commur.ity and especially the responsibility to be ready to take on new, but 
related sets of tasks. Below, we describe and evaluate an approach to 
training that pursues this particular variant of the constructivist theme. 

For the last ten years, we have worked nth a team of colleagues from 
the University of Pittsburgh Learning Reser.r-h and Development Center and U.S. 
Air Force Armstrong Laboratory to develop an instructional approach we call 
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intelligent coached apprenticeship^ (Eggan & Lesgold, in press; Katz & 
Lesgold, 1991, in press; Katz, Lesgold, Eggan, & Gordin, 1992, in press; Gott, 
1987; Gott, 1989; Gott, Hall, Pokorny, Dibble, & Glaser, 1992; Gott, Pokorny, 
Alley, Kane, & Dibble, in press; Hall, Gott, & Pokorny, in press; Kane, 1993; 
Lajoie & Lesgold, 1989; Lesgold, in press; Lesgold, Eggan, Katz & Rao, 1992; 
Glaser, Lesgold, & Gott, 1986; Lesgold, Katz, Greenberg, Hughes & Eggan, in 
press; Lesgold & Katz, 1992; Lesgold, Lajoie, Bunzo & Eggan, 1992; Nichols, 
Gott, Alley, & Pokorny, in press; Pokorny & Gott, in press). This approach is 
based upon the opportunity to experience the most difficult aspects of 
cognitively-intense jobs in a simulated work environment where assistance, in 
the form of an intelligent computer-based coach, is always available and where 
there are opportunities to reflect on simulated work experiences. We have 
developed two generations of tutors for training a specialized electronics 
maintenance job in the U. S. Air Force, namely the F-15 manual avioi.ics test 
station specialty (see Appendix I for an overview of the F-15 Manual Avionics 
Test Station job). The \;utor focuses on the hardest part cf the job, 
isolating failures in the test station itself. Both generations of the 
training system we have built, named Sherlock 1 and Sherlock 2, have worked 
remarkably well, in terms of success in fostering high levels of job expertise 
and, with Sherlock 2, promoting transfer to new electronics troubleshooting 
tasks on novel equipment. 

The primary activity within Sherlock is holistic work, at the highest 
levels of real -world difficulty, though often this requires coaching, which is 
available on demand. The approach has several distinguishing characteristics: 

- Learning activity is centered in a simulated work environment. 

- Learning activity is centered around problems that exemplify the 
hardest parts of the job for which one is being trained (problems defined 
in collaboration with master technicians on the job). 

For each problem, two kinds of activities occur: 

- The student solves the problem, requesting advice from the 
intelligent tutor/coach as necessary. 



^Sherlock 2, the current embodiment of the ideas we discuss in this paper, 
has been a collaborative effort that has included Daniel Abeshouse, Marilyn 
Bunzo, Roberta Catizone, Dennis Collins, Richard Eastman, Gary Eggan, Mark 
Gallaway, Robert Glaser, Maria Gordin, Sherrie Gott, Linda Greenberg, Ellen Hall, 
Edward Hughes, Ron Kane, Sandra Katz, Dimitra Keffalonitou, David Kieras, Susanne 
Lajoie, Alan Lesgold, Robert Linn, Thomas McGinnis, Johanna Moore, Dan Peters, 
Bob Pokorny, Rudianto Prabowo, Govinda Rao, Rose Rosenfeld, Kurt Strobel, Gary 
Walker, and Arlene Weiner. Collins, Gallaway, Gott, Hall, Kane, Pokorny, 
Strobel, and Walker are U.S. Air Force uniformed or civilian employees; David 
Kieras is at the University of Michigan; Robert Linn is at the University of 
Colorado; the others are or were at the University of Pittsburgh. 



- The student reviews a record of her^probl em- solving activity, 
receiving constructive critique from the coach. 

Sherlock provides a simulation of the work environment for the F-15 
avionics job, using a combination of video and computer graphic displays. 
Simulated controls can be operated with the computer mouse, and the displays 
change to reflect an underlying computer simulation of the devices being 
simulated. Since the fundamental activity of troubleshooting in this job is 
making tests with meters, this is provided realistically by having icons of 
meter probes that can be "attached" to video images of device test points. 

To complement coached learning by doing, we have developed a collection 
of tools for post-performance reflection. One provides an intelligent replay 
of the trainee's actions. A trainee can "walk through" the actions he just 
performed while solving the problem. In addition, he can access information 
about what can in principle be known about the system given the actions 
replayed so far (the work of troubleshooting is mostly the making of 
electrical measurements and then figuring out which possibilities are ruled 
out and which supported by the pattern of results). Also, he can ask what an 
expert might have done in place of any of his actions, get a critique of his 
action, and have his action evaluated by the system. In addition, extensive 
conceptual knowledge about the system's functions is available from 
intelligent hyper graphic displays of an expert's circuit model schematic 
drawing. In these drawings, the boxes that stand for circuit components are 
all mouse-sensitive and can "tell about themselves." We have also built a 
tool for displaying an expert solution to the problem, again with extensive 
conceptual information available as appropriate to each step. Further, there 
is an option for side-by-side listing of an expert solution and the trainee's 
most recent effort. 

The tools we have built are motivated by substantial research on the 
reflective activities that might foster learning. For example, Chi and Van 
Lehn (1991; Van Lehn, Jones & Chi, 1992) analyzed the activity of more and 
less effective learners in studying worked-out physics problems. They found 
that more effective learners showed a different pattern of study, paying more 
attention to the conditions under which various steps in the solution were 
taken, to the relations between actions and goals, to the consequences of 
actions, and to the underlying meanings for formalisms such as equations. 
Gott, Hall, Pokorny, Dibble, and Glaser (1992) made similar observations in a 
study where transfer of skill under naturalistic conditions was investigated. 
Bielaczyc, Pirolli, & Brown (1993, April; Pirolli & Bielaczyc, 1989) 
demonstrated that students could be taught a similar approach. 

Working from the Chi, Bielaczyc, and Gott studies just cited, we can 
infer several possible roles for post-problem reflection. First, if the 
trainee reached impasses during his efforts and had to ask for help, then 
there is some learning work to be done. The trainee must figure out why the 



^To enhance readability, we alternate between masculine and feminine 
pronouns rather than using more cumbersome forms. About 23% of the target 
population of the Sherlock system are women. 
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suggestions of the intelligent coach were useful and what rule(s) can be 
inferred. Second, problem solving experiences afford opportunities for tuning 
the generality of procedural knowledge and also for elaborating conceptual 
knowledge. This is especially the case where intuitive guessing was part of 
the solution process: 

"I tried doing X because it seemed like it might work; why did I think 
it should work?" 

Problems can often be solved in non optimal ways. When this happens, 
there is no impasse to cue the trainee that his knowledge needs further 
tuning. So, criticism may be a useful part of the reflection opportunity. Of 
special relevance are the trade-offs involved in testing hypotheses by 
swapping parts versus measuring electrical properties of the faulted system. 
Just ai in football, part of what a coach can do is to point out possibilities 
for improvement that may not be evident to the trainee with respect to cost- 
benefit trade-offs in the selection of solution steps. 

While this instructional approach differs radically from the approaches 
promoted by traditional instructional design schemes, it is equally dependent 
upon good task analysis. What is different is that the structure of learning 
tasks is more authentic, rooted in the needs of practice (or simulated 
practice) rather than being derived directly from task analysis structure 
(Gott, 1987; Hall, Gott, & Pokorny, in press). 

One important component of our constructivist approach is the intelligent 
hyperdi splay. When Sherlock constructs a schematic diagram to help illustrate 
the advice it is providing, that diagram is organized to show expert 
understanding about the system with which the trainee is working. The 
structure of the diagram reflects the expert representation of the circuitry 
involved in carrying out the function that failed, as revealed in a detailed 
cognitive task analysis. What is displayed is approximately what a trainee 
would want to know at that time, but every display component is "hot" and can 
be used as a portal to more detail or explanation. The part of the system on 
which the expert would be focusing at a given point in the problem solution 
process is allocated the most space in the diagram and presented in the most 
detail. All diagram components are "buttons" that can be pushed to expand 
their level or detail. Boxes in the diagram are color coded to indicate what 
is known about them given tests carried out so far. Circuit paths are color 
coded to indicate whether the electrical properties of those paths are known 
to be appropriate or inappropriate for the function that has failed. Sometimes 
during problem solving, information is deleted from the display before it is 
shown, so that the trainee doesn't substitute looking at labels in the 
displays for inferring what circuitry is involved in the functional failure 
being diagnosed. 

Tutoring for Transfer 

Our principal pedagogical goal in initiating the Sherlock project was to 
accelerate the development of complex, technical problem solving skills. In 
Sherlock 2, this goal was accompanied by a second, equally important one, 
namely, to foster technical adaptiveness. Skill flexibility is vital in ;in 



era of rapid technological change. Earlier work conducted by our research 
group in the area of transfer revealed some important findings about 
intentional transfer, or the undisguised requirement to transfer one's 
existing knowledge and skill to novel tasks (Gott et al, 1992). Those data 
revealed that mental models of devices exert strong influence on knowledge 
access, additional learning (knowledge extension), and subsequent diagnostic 
reasoning. In the domain of avionics troubleshooting, the primary content of 
transfer takes the form of abstract knowledge representations. Time and again 
we observed good learners access theiv- existing mental models of equipment 
structure and function and their schema of the troubleshooting task (Figures i 
and 2). They then used these models as flexible blueprints to guide their 
performance as they crafted solutions to new problems. Their prior models 
became interpretive structures, and when these models were inadequate, better 
learners flexibly used them as the basis for transposed snd elaborated 
structures that could accommodate the novel situations. They were ready and 
willing to construct new knowledge that was gounded in their existing 
representational and functional competence. 
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*Note: LRU stands for Line Replaceable Unit, a module removed 
from the aircraft on the flightline. 

Figure 1. Top-level mental model of avionics eqipment system 



1. EUIE OUT IMTT UNDER OEST (UOT) AND TEST PACKBGE 

Find out vAiether problem is in the UOT or the test equipcnent (i.e. , Test Package and 
Test Station) . Note: Once cxjnnected to the Test Station, the Line Replaceable Unit (1511) 
beconvss the unit under test 

2. INVESTIGKIE lEftSORBMENT SIGNMj PMU 

Find out the location of the problem on the measuroatient signal path — either a bad 
conponent or one receiving bad data. 

3 . INVESnCftTE MEASUREMENT DKEA PMH 

Find the component in the measurement data path that is causing the bad control date 

ingat. 

4 . INVESTIGATE CW3SE OF INOCSBaRECr UGHT SmOJS (CN ERCWT PANEL) 

Find the coitponent causing the incorrect li^t state of the indicator which is in 
the measurement or stimulus data path. 

5. INVESnCAIE STTMOniS SIGNAL PASH 

Find out the location of the problem on the stimulus signal path. 

6. INVESTIGATE STMILUS DAIIA Efifffl 

Find the location of the problem on the stimulus data path. 
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Figure 2. Schema of Troubleshooting Task for F15 Test Stations 
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By contrast, less able performers devised ways to avoid this adaptive 
learning experience. They displayed maladaptive behaviors as they 
oversimplified new problems and overgeneral ized existing structures. As a 
result, their performance in the new domain appeared novice-like, without the 
benefit of abstract plans and adapted models. They were wedded to their old 
structures, unable to perceive that the functional variations in the devices 
in the novel domain were plausible extensions of their current understanding. 

These findings in turn influenced instructional design decisions in 
Sherlock 2, as follows: (a) high quality device models were fostered with 
liberal scaffolding in the form of illustrative equipment diagrams used in 
coaching and reflective followup; (b) interactive video representations of 
actual electronic devices dominated the learning environment (in 
constructivist terms, the video devices provided phcnomenaria to be 
manipulated and otherwise exercised to build understanding); (c) in the 
tutor's coaching (or Information Banks) general functional terms were used to 
describe the electronic devices and tests, not problem specific terminology; 
(d) general terminology was also be used to characterize the goal structure 
that typifies "the (generic) plan" for troubleshooting in this domain; and (e) 
the reasons behind preferred goals and procedures were made explicit to reveal 
the expert's cost-benefit reasoning in evaluating alternative courses of 
action. 

These design decisions were implemented in Sherlock 2 as follows: 

- Equipment diagrams appear frequently in the tutor to illustrate the 
coaching (an Information Bank) that is available to students as well as the in 
the reflective followup activities at the end of a troubleshooting session 
(see Figure 3). 

- Functional terminology that reveals the electronic test that is being 
run is consistently abstract in nature, not specific to the given 
troubleshooting (scenario) situation being diagnosed (see Figure 4). 

- The reasons behind procedural and strategic steps are now made 
explicit (see Figure 5). 

- System or "How It Works" coaching is presented at both general and 
specific levels of detail (see Figure 6). 

- A standard troubleshooting goal structure is used as a core plan in 
"How to Test" coaching across all scenarios (see Figure 7). 

- In the Reflective Followup, the cost-benefit reasoning behind the 
illustrative expert solution is made explicit (see Figure 8). 

- Also, in the Reflective Followup, general troubleshooting principles 
are emphasized during the dialog between student and coach (see Table 1). 



Table 1. Principles of Good Troubleshooting 



1. Perform test procedures safely. 

2. Accurately identify active equipment components to investigate. 

3. Conduct measurements accurately (correct device, correct signal type, 
correct probe placement). 

4. Test components thoroughly. 

5. Swap only after verfying component is faulty. 

6. Investigate functional areas of equipment in logical sequence. 

7. Make accurate inferences about malfunction based on test results. 
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Anuming the Circuit has only on* 
fault 

TlM input signal at pin 21 and pin 20 is 
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PlB«se select a different componertto 
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Figure 3. Illustration of Equipment Diagram Used in 
Reflective Followup Activity 
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CIRCUIT LEVEL COACHING: 

There are two main starting points for tracing the measurement 
signal path: the UUT Test Package (TP) colximn of the checkout Tech 
Order, and the switch settings on the RAG drawer. 

From the UUT TP column; 

1. Look at the UUT TP column to find out which UUT pins are 
being tested. 

2. Examine the test package schematics to identify the test 
package plug and pins that are connected to these UUT pins. 

3. Trace from the active plugs and pins on the test package 
to the active jack and pins on the RAG drawer. 

4. Use the jack/ switch index located in front of the RAG 
schematics to find the zone in the RAG schematics where these 
active RAG jack and pins are located. 

5. Go to this zone, and trace the rest of the measurement 
signal path from the active RAG jack and pins to the measurement 
device . 



From the Switch Settings on the RAG Drawer: 

1. In the checkout Tech Order look at the settings. 
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Figure 4. General terminology used in coaching regarding 

electronic test being run 
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step fro» . Master, s solution trace as axmotate. in t.e Refieotive 
Followup: 

TO find out whether the problem is in the UUT or the 
test equipment; 



GOAL: 
PROCEDURES! 



The Master technician: 

- replaced the OTT's A4 card; reran test; still 

- Wiethe following test at Jac)c 1 (Jl) 
llf.TT.r^^s\ -Thi^las^^thre^ecSd Tesult^ 



Figure 5. Reasons Behind Procedural Step Made Explicit 
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TEST POINT SELECTION RELAY CARD - HOW IT WORKS 



Level 1: 



Level 2: 



Level 3: 



The RAG'S A1A3A13 card is a relay card. In a relay 
card, data signals are applied to the coils of the 
relays on the card in order to set (or reset) the 
relays. In this way, one or more paths are created 
through the card so that signals can travel from the 
card input(s) to the output ( s) . 

The RAG'S A1A3A13 is a Test Point Selection relay 
card. A Test Point Selection relay card consists of 
ten relays and all ten relays share a common output 
but have different inputs. In this way, a Test 
Point Selection card is designed to create only one 
path between its output and one of its ten input pin 
pairs. This means that one Test Point Selection 
card is capable of creating ten different relay 
circuit paths, but should have only one relay active 
for any one T.O. test. 

To work properly, the RAG's A1A3A13 should have only 
one of its ten relays selected at any one time. 

For this test step. Switch 53 is set to 0, Switch 52 
is set to 0 and Switch 54 is set to B. This means 
that 28VDC should be present on pin 33, OVDC on pin 
30, and relay BOO should be set. 

When relay BOO sets, a path is created between input 
pins 51 and 52 and output pins 11 and 12. 
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Figure 6. "How It Works" coaching at general and specific levels. 
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1. Rule out UUT and Teat Package first. 

2. investigate measurement signal path next. 

3. investigate measurement data path next. 

4. Find source of incorrect light state. 

5. investigate stimulus signal path. 

6. Investigate stimulus data path. 



Figure 7. Standard Goal Structure Used in Coaching 
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Illustrative Explanation from a Master: 



After verifying that the UUT and Test Package are good, you 
now know that the problem is in the Test Station. The measurement 
signal path in the station, i.e., the path between the UUT and the 
station's Digital Multimeter (DMM) , is a good place to start. 
Testing the signal path allows you to investigate a smaller and 
somewhat easier functional area of the station, and so you can 
constrain your search in an effective way. 

If you bypass the measurement signal path and start in the 
measurement data functional area, you have a larger area with more 
difficult components to troubleshoot . The measurement data 
functional area includes logic cards and switches, for example. 
The components that send control voltages to relay cards on the 
measurement signal path are among the components located in the 
measurement data functional area. 




Figure 8. Cost-Benefit Reasoning Behind Master Solution Step 
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Purpose and Rationale for Tutor Evaluation Study 



With these major instructional design decisions having been implemented, 
the goal of the Sherlock 2 evaluation study was to determine if an intelligent 
tutoring system that is informed by detailed cognitive models of 
troubleshooting performances is effective in both accelerating skill 
acquisition and fostering adaptive expertise. A controlled experiment was 
conducted that involved U.S. Air Force technicians at three geographically 
separated F15 flying wings. 

Trainees were evaluated on a number of technical proficiency and 
experience indicators to establish matched experimental and control groups. 
The principal form of assessing troubleshooting skill was a type of 
structured, thinking aloud verbal protocol, called a Verbal Troubleshooting 
Test (VTT). The advanced, most experienced technicians in the shops were also 
tested on VTTs and other assessment instruments in order to establish the 
upper limits of performance for this job. 

It was hypothesized that the experimental group would demonstrate an 
accelerated rate of skill acquisition compared to the control group trainees 
in moving toward the level of performance displayed by advanced technicians. 
These expectations were predicated on the following premise: in a learning 
environment that provides direct but coached problem solving experiences and 
one in which cognitive skill components and processes have been precisely 
identified as instructional targets, the acquisition of complex skills such as 
electronic troubleshooting can be speeded up. The expected accelerated 
acquisition would be attributable to (1) the better instructional content 
enabled by cognitive models that make the unobservable facets of 
troubleshooting knowable by learners and (2) better methods where knowledge is 
tied to its uses in the world and learning is supported by direct manipulation 
experiences (phenomenaria) , coaching, modeling, and other scaffolding embedded 
in the computer tutor. 

It was further hypothesized that the experimental group would 
demonstrate adaptiveness in their newly acquired troubleshooting skills 
compared to the control group when tested on a novel equipment system. These 
expectations were predicated on several premises. First, a learning 
environment where extensive practice is available to trainees would build up 
robust task schemas supported by conceptual support knowledge that explains 
the "reasons why" tasks are structured the way they are. Thus, knowledge 
structures would be both robust and flexible. Secondly, all coaching and post- 
session reflective feedback would provide general as well as task specific 
explanations to inject elasticity into system, procedural, and strategic 
knowledge components. 

METHOD 
Subjects 

Forty-one Air Force apprentice and thirteen master technicians, for a 
total of 54 F15 avionics maintenance technicians, participated in the 
evaluation study. Assignment to either the apprentice or master group was 
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based on VTT proficiency scores as well as on other pertinent factors, such as 
aptitude and experience. Technicians in this specialty work in a shop 
environment, vs. on the flightline. Their most challenging tasks involve 
troubleshooting the test stations that simulate the jet for purposes of 
testing components (black boxes) that are removed from the aircract (by 
flightline technicians) because they are presumed faulty. Subjects were 
selected from three geographically separated sites -- Langley AFB VA, Nellis 
AFB NV, and Eglin AFB FL. Table 2 shows the distribution of subjects across 
the three sites. The apprentice technicians had an average of 33 months 
experience on the job, while the master technicians averaged 124 months (10 
years 4 months) experience. 

Table 2. Number of Subjects by Groups Across Sites 



Sites 

Group Lanolev AFB VA Nellis AFB NV Eglin AFB FL 



Experimental 


6 


12 


0 


Control 


4 


1 


18 


Master 


2 


4 


7 


TOTAL 


12 


17 


25 



Instruments 



Four different instruments were used in the Sherlock 2 evaluation. Two 
were assessments of learning: Verbal Troubleshooting Tests (VTTs) and 
Noninteractive Troubleshooting Te':ts (NITs); one was a Background 
Questionnaire to document the subjects' personal data; and one was a Tutor 
Report Card, which tutored airmen used to evaluate various dimensions of the 
tutor as an instructional system. We will only treat the VTTs and NITs here. 
See Gott, Pokorny, Alley, Kane, and Dibble (in press) for a complete 
description of all instruments. 

Verbal Troubleshooting Tests (VTTs) 

An accepted principle in practice-oriented (vs. academic) instruction is 
that both training and testing should mirror the criterion performance (Gott, 
1987). This principle applies to modern cognitive apprenticeships as well; 
however, the mirroring process is complicated by the fact that performances in 
such apprenticeships are more mental than physical. No longer is it effective 
to focus on (diminished) overt behaviors and observable end products for 
either instructional or assessment purposes. Internalized cognitive processes 
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and structures are the legitimate targets of assessment. For these reasons, 
we developed special purpose instruments to evaluate the Sherlock tutors. To 
acce.s and measure the covert processes and structures that Sherlock targets 
(an( that, we would argue, really explain competence), we developed Verbal 
Troubleshooting Tests (VTTs) as the principal measure to evaluate tutor 
effectiveness (Gott, 1987; Gott et al , in press). The VTTs satisfy the 
principle of mirroring the criterion performance and can accommodate the 
assessment of internalized cognitive processes and structures. The general 
form of the VTT closely resembles the structured thinking aloud verbal 
protocol process used in conducting the cognitive task analysis for Sherlock 2 
(Gott et al, in press) (see Figure 9). The diagram in the figure is drawn by 
the subject to illustrate the solution step and thereby reveal the mental 
model he is using. 



PRECURSOR 

I wMt to ••• If th« LRU ID r««i«ter i« so««. 



ACTION 

R«Me«« th« ealtl* fr«M Ji2 of tii* LRU »ti4 out th« 
path throvgh th« LRU frOM pin •• t« pin 12*. 

RESULT 

Th« r«*dln« I* l.M M«hM«. 



INTERPRETATION 

Th« prebMM Isn't in ih« LRU. it's in th* i««t •isilon 
or th« M«t paekac*' 







J12 








12S 








< LRU 10 

< RESiSTON 


SB 



















TEST STATION TEST PACKAQl 



Figure 9. One Solution Step Illustrating the 

PARI Cognitive Task Analysis Methodology 
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The basis for the VTT is a structured example of a verbal protocol 
technique. It is structured in the sense that subjects are queried according 
to a standard framework in addition to being asked to think aloud in general. 
The imposed structure reflects the recursive nature of diagnostic reasoning: 
hypothesize, test (collect data), interpret data,, hypothesize, test, 
interpret, and so forth, until a solution state is reached. 

The most valued feature of the VTT is that it allows a close 
approximation of hands-on performance, without the costly and inefficient 
utilization of actual equipment. In the Sherlock project, the VTTs were 
designed to represent the domain by covering critical areas identified by the 
cognitive task analysis. The CTA results produced a functional model of the 
equipment configuration that was shared by expert troubleshooters {refer to 
Figure 1). We in turn have considered this equipment model and the actions 
used to investigate it to be "the domain" to be covered instructionally. Tha 
basic functionality works as follows: in a simulation of in-flight operation, 
the stimulus generators inside the test station produce signals that are 
routed via the Relay Assembly Group (RAG) drawer, through the test package, to 
the UUT. Signals returning from the UUT pass through the test package and RAG 
drawer on their way to a measurement device in the test station. The 
technician monitors the outputs of the measurement devices and compares the 
actual readings with the ,xpected readings listed in a check-out tech manual. 
The manual lists the expected values as well as the probable source of fail 
for each test that is run on the UUT. 

Avionics experts with considerable test station troubleshooting 
experience used the CTA results and Sherlock I findings to select or develop 
the VTTs for Sherlock 2. There were two pretest VTTs and two posttest VTTs. 
During data analysis, one pretest VTT had to be eliminated because it covered 
content in the tutor that was deleted from the treatment because of time 
constraints in the field. 

Administration . Each pretest and posttest VTT was individually 
administered by an experienced F15 avionics expert from our research team. 
The examiner instructed each subject to troubleshoot in the same way as she 
would in the actual work environment on the real equipment. In each VTT, the 
examiner posed a situation to the technician where a fault had occurred during 
the routine benchchecking of an LRU. The situation for each VTT corresponds 
to a Problem Statement similar to that depicted in Table 3. The examiner 
begins by asking, "What is the first thing you would do in the shop if this 
happened?" and the technician responds with an action, thereby beginning the 
recursive diagnostic reasoning process. The step-by-step solution process 
engaged in by the subject is very similar to troubleshooting the actual 
avionics equipment. A series of recursive action-result steps unfolds and 
constitutes the solution (Steps 1-7 in Table 4.) Ninety minutes were allowed 
for completion of each VTT. 

Table 3. Exemplar VTT Problem Statement 



While running a Video Control Panel Unit, Test Step 3e fails. The 
panel lamps do not illuminate. All previous test steps have passed. 
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Table 4. Exemplar VTT Solution Trace 

Problem statement: While running a Video Control Panel unit. Test Step ..e 
fails. The panel lamps do not illuminate. All previous steps have passed. 



Step 1: 

What would you do first if this 
happened in the shop? 

Result: Test still fails 



Action: Reenter Step 3e 
instructions 



Step 2: 

What would you do next? 
Result: Test still fails 



Action: Swap UUT with shop 
standard; rerun test 



Step 3: 

What would you do next? 



Result: Test still fails 



Action: Swap in a 

known good test 
package; rerun test 



Step 4: 

What would you do next? Action: Take an AC voltage 

test at output of Manual 
Stimulus Assembly 

Result: 0 volts AC 



Step 5: 

What would you do next? Action: Take a DC voltage 

test at data flow into 
Card A22 

Result: 28 Volts DC 



Step 6: 

What would you do next? 



Result: 1999K ohms 



Action: Make ohms test from 

relay card input to out- 
put 



Step 7: 

What would you do next? 
Result: Test passes 



Action: Swap card A22; 
rerun test 
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Scoring . To handle the scoring demands of nonstandard data such as VTT 
protocols, we developed an objective scoring i>j"Jtem for Sherlock 1 using 
policy capturing techniques on judgments which wert generated by experts as 
they evaluated VTT solution traces (Gott, 1987; Pokorny & Gott, in press). 
From this data base cf explicated judgments from multiple experts, it was 
possible to extract a scoring policy that can be defined by seven principles 
of good troubleshooting (see Table 1). These principles provide the 
foundation for the detailed scoring rules that constitute the complete scoring 
system (Gott et al , in press). These general principles figure prominently in 
the assessment system used in Sherlock 2. 

For Sherlock 2, the interrater reliabilities (based on holistic VTT 
scores) for the three tests were .96, .95, and .94. For all three 
correlations, the 1-tailed significance level was £ <.O01. 

Accompanying materials. In accordance with our goal of ensuring that 
Sherlock is a natural extension of the technician's real work environment, 
there are two standard pieces of technical material that are needed to work 
the VTTs as well as perform maintenance tasks on Sherlock: LRU Check-out 
Procedures and Test Package/Test Station schematic diagrams. The Sherlock 
analog documents closely mirror the real world tech data. There is also a 
User's Manual to assist the technician in operating the computer tutor. 

Noninteractive Troubleshooting Test 

The Noninteractive Troubleshooting (NIT) test (the second instrument to 
be discussed here) is a paper-and-pencil instrument designed to complement the 
VTT as a measure of troubleshooting skill. It was developed for two reasons: 
(1) to provide a basis for a convergent measure of troubleshooting proficiency 
(with the VTTs) and (2) to examine alternative hypotheses (aside from the 
Sherlock intervention) that could plausibly explain the expected pre- to 
posttest VTT gains. This test differs from the VTT in several ways: first, 
it is a written test administered under highly standardized conditions with a 
limited role for the examiner, and second, questions require short, 
unambiguous, written responses, even though the realistic troubleshooting 
context associated with the VTT is preserved as much as possible. Parallel 
forms were developed for the NIT Pretest and the NIT Posttest. For untutored 
subjects, pre and posttest NIT scores were significantly correlated (r = .76, 
one-tailed significance £ = <.001). 



Other Instrumentation and Materials 

The Novel Equipment System 

To investigate the generality of troubleshooting skills acquired by 
subjects tutored on Sherlock 2, an avionics expert on- our research team 
created a mythical equipment system called Frankenstation (Kane, 1993). 
Frankenstation is an automatic test station, not the type of manual station 
represented in Sherlock. Its function is to test components of the Integrated 
Ground Operational Radar (IGOR) system. To achieve that purpose, it has the 
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same functional areas that are common to any avionics test station (see Figure 
10). Its primary uniquc-.ess (vis-a-vis Sherlock) is that it is automated, 
meaning it is computer (not human) controlled. Therefore, Frankenstation tech 
data includes programming information for the ATLAST programming language, for 
example, which would be novel to manual test station maintenance personnel. 
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Figure 10. Top-level Functional Diagram of Frankenstation 

Note: *Functional terms such as stimulus, measurement, and routing were not 
provided to subjects on this diagram. They had to consult other 
technical data to make those functional determinations 
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RESULTS 



Comparability of Groups Prior to Intervention 

Initial comparisons of the experimental and control groups on the VTT 
and NIT pretests and on other performance-related measures (aptitude and 
experience) revealed no statistically reliable differences prior to the 
intervention (Table 5). This indicates that the groups are equivalent for 
purposes of this evaluation study. 



Table 5. Pretest Measures of Aptitude, Experience, and 
Troubleshooting Proficiency 



Troubleshooting 
Aptitude Measures Experience Proficiency 
Group N Electronics AFQT Manmonths VTT NIT 



Novices 
Control 23 

M 81 75 27 57 65 

SD 10 16 25 36 12 

Expermntl 18 

M 84 71 39 39 64 

SD 10 14 33 66 20 

Masters 13 

M 87 81 124 89 79 

SD 11 14 31 9 8 



Comparisons of the Novice groups with the Master group revealed expected 
differences in experience (t = -9.76, fi 0.000) and troubleshooting 
proficiency (VTT Pretest: t = -2.81, fi = .007; NIT Pretest: t = -3.03, fi = 
0.004). There were no statistically reliable differences (Novices, vs. 
Masters) on the two aptitude measures. 

The relationships of VTT and NIT Pretests with the other performance- 
related measures (aptitude and experience) are shown in Table 6. The 
strongest correlations were between (1) the two aptitude measures (Electronics 
and AFQT), (2) one aptitude measure (Electronics) and the NIT Pretest, and (3) 
the two troubleshooting pretests (VTT and NIT). The only non-significant 
correlation was between aptitude (AFQT) and experience. 
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Table 6. Intercorrelations of Pretest Measures 



Aptitude Experience Proficiency 
Measure AEQI Manmonths VTT NIT 



Aptitude .60** .32* .43** .49** 

(ASVAB Elec) 

Aptitude .28 .41** .36* 

(AFQT) 

Experience .40* .42** 

(Manmonths) 



VTT Pretest .46 



"fi <.01 **fi <.001 



** 



Post Intervention Results 

Experimental and Control Group Comparisons 

As predicted, VTT and NIT posttest scores revealed large and 
statistically significant differences in favor of the experimental group over 
the controls (Table 7): VTT 3 (t = -4.04, fi = 0.000); VTT 4 (t = -3.72, fi = 
0.001); NIT (t = -2.77, fi = 0.009). 
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Table 7. Posttest Measures of Troubleshooting Proficiency 
(Sherlock 2) 



Sherlock 2 Posttests 



Group N VTT 3 VTT 4 NIT 

Novices 
Control 23 

M_ 59 58 75 

SD 37 37 H 

Experimental 18 

M_ 95 91 87 

SD 5 7 12 

Masters 13 

M_ 85 86 86 

SD 12 11 11 



The single holistic indicator of VTT performance can be decomposed by an 
analysis of the components of troubleshooting that are embedded in verbal 
protocols and known to be associated with expertise, per the cognitive task 
analysis. Such components include measuring to investigate the equipment 
rather than swapping equipment parts, systematicity in investigating circuitry 
versus random actions, end efficiency in targeting areas to investigate by 
optimizing the information value-to-cost ratio associated with each potential 
action. Results of the componential analysis are reported in Gott, et al (in 
press) . 

Also, as predicted, the Frankenstation VTT and NIT posttest scores 
revealed large and statistically significant differences in favor of the 
experimental group over the controls (Table 8): VTT (t = -2.93, £ = 0.006); 
NIT (t - -2.34, E = .025). 
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Table 8. Posttest Measures of Tr"nsfer 
(Frankenstation) 



Frankenstation Posttests 
Group N VII NIT 



Novices 
Control 21 

M_ 55 72 

SD 31 4 

Experimental 17 82 80 

M_ 23 10 
SD 

Masters 12 

M_ 91 88 

SD 22 12 



The effect size for each of the posttest measures is shown in Table 9. 
As a basis for comparison, the average effect size for new science and math 
curriculum in U.S. schools is reported to be .3 sigma (Bloom, 1984). 



Table 9. Effect Size for Posttest Measures 



Group 







Control 






Experimental 


Effect 


Measure 


N 


M 


SD 


N 


M 


SD 


Size 


VTT 3 (Sherlock) 


23 


59 


37 


18 


95 


5 


1.27 SDS 


VTT 4 (Sherlock) 


23 


58 


37 


18 


91 


7 


1.17 SDS 


NIT (Sherlock) 


23 


75 


14 


18 


87 


12 


.87 SDS 


VTT (Frank' tn) 


21 


55 


31 


17 


82 


23 


.96 SD5 


NIT (Frank'tn) 


21 


72 


11 


17 


80 


10 


.76 SDS 
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Qualitative Analyses of Frankenstation VTT Data 



The single holistic indicator of Frankenstation VTT performance can be 
decomposed into discrete components of troubleshooting skill that are embedded 
in the verbal protocols and known to be associated with expertise, per 
cognitive task analysis data. Such components include (1) thoroughly 
measuring suspect equipment components during troubleshooting rather than 
swapping equipment parts prematurely, (2) using automated diagnostic aids 
(such as system self-tests) in a targeted, efficient manner, (3) following a 
logical, efficient strategy in troubleshooting, and finally, (4) isolating the 
fault and thereby solving the problem within the time allowed. 

Table 10 shows the comparative swapping actions of the Master, 
Experimental, and Control Groups. Instances of swapping without testing 
clearly differentiated the untutored airmen (Control Group) from the Tutored 
and Master technicians. Table 11 shows how the three groups used self -tests 
(ST) (diagnostic software) in the course of troubleshooting. While the three 
groups used the self-test capability at the same relative frequency, the 
Contrr Group clearly differed in its manner of implementation. Of the 16 
occasions when control subjects used self-tests, there were 11 times when they 
used the ST inefficiently. They chose to run the ST from the beginning until 
a fail was encountered as opposed to selecting self -tests that targeted the 
test station devices that were being used in the failed test. Conversely, 
Master and Tutored airmen used the ST selectively because they could identify 
from the program listing and the test station schematics which test station 
components were being used during the failed test. 

Table 10. Frequency of Component Swapping Without Complete Testing 

(Frankenstation VTT) 



Swaps 



Swaps 



Group 



No Testing 



Partial Testing 



Control s 



50 



12 



Experimental 



3 



13 



Masters 



8 



15 



25 
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Table 11. Frequency and Quality of Self -Test Use 
(Frankenstation) 



Group Self Test Use Frequency of Inefficient Use 

Controls 16 11 

Experimental s 12 4 

Masters 14 2 



Instances of following the logical, most effective strategy in 
troubleshooting Frankenstation clearly favored the Tutored airmen (Table 12). 
And finally, the Tutored and Master subjects bettered the Controls in terms of 
isolating the faulty component in Frankenstation i.e., solving the problem, in 
the time allotted (Table 13). 

Table 12. Number of Violations in Logical Sequence of Troubleshooting 

(Frankenstation) 



Group Number of Violations 

Controls 21 
Experimental 9 
Masters 20 



Table 13. Percentage of Solutions to Frankenstation VTT by Group 

Group Percentage Achieving Solution 

Controls 63.6 

Experimental 70.6 

Masters 83.3 
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Discussion of Frankenstation Results 



There are a number of intriguing perspectives to adopt in considering 
the transfer effects from Sherlock 2 to Frankenstation; however, space 
limitations permit us to explore briefly only three types of empirically 
supported effects, all of which are related to the findings in Tables 10-13. 
When we examined the Frankenstation VTT protocols qualitatively, evidence of 
the transfer of system, procedural, and strategic knowledge was clearly 
present. 

Transfer of System and Strategic Knowledge 

The use of self-tests and excessive swapping reveals the differing 
device models which underpin the actions of Tutored and Untutored airmen. 
Consider the following solution steps generated by an Experimental and Control 
subject after each had eliminated the UUT and Test Package from suspicion in 
Steps 1 and 2: 



Experimental 

Step 3: Run self -test on 
the stimulus device. 
Result: passes 



Step 4: Run self -test on 
the measurement device. 
Result: passes 

Step 5: Run self-test on the 
control /routing device 
Result: fails 



Control 



Step 3: Run self -test from 

the beginning. 

Result: control /routi ng 

device fails; calls r-.it 3 relays 

as possible causes 

Step 4: Replace all 3 
relays; rerun self -test 
Result: same fail 



Steps 6, 7, & 8: Isolate fail within 
the control /routing device by 
measuring at test points that 
will isolate malfunctioning 
relay; swap malfunctioning 
relay; rerun self -test 
Result: passes 

These data suggest that the Experimental subject is working from the standard 
device model emphasized in Sherlock 2 (see Figure 1). He looks for and locates 
devices in the novel erjipment that serve the stimulus, measurement, and 
control/routing functionalities, in effect instantiating the general schema 
represented in Figure 1. The Untutored airman, by comparison takes an inefficient, 
"default-type" step that could take hours, i.e., she runs the self-test from the 
beginning, with no particular device targets in mind. Using the results of the 



27 



wholesale self-test, she then swaps the three relays, only to learn that the 
call outs from the self -test do not fix the problem (not an uncommon outcome). 

By comparison, the Experimental subject takes measurements, which 
provide much more information than swapping, and efficiently localizes the 
malfunctioning relay before swapping any components. The Tutored airman 
appears to have generalized device and strategic knowledge from Sherlock. 
This comparison is not an isolated example, but rather the prototypical 
actions taken by Untutored subjects, as reported in Tables 10 and 11. 

Transfer of Procedural Knowledge 

Another qualitative inspection of two solution traces from a Tutored and 
Untutored subject reveals both a specific procedure and a global principle 
that tutored airmen appear to have generalized from Sherlock 2. 

Experimental 

Step 5: Rerun UUT; measure ohms from pin to pin on suspect relay. 
Result: 0 ohms 

Step 6: Measure ohms from pin to pin on different relay. 
Result: open 

Step 7: Measure control voltage inputs to faulty relay identified in 
Step 6 

Result: 5 vpp 

Steps 8 & 9: Measure remaining control voltages to suspect relay. 
Control 



Step 4: Run self-test from beginning 

Result: Control/routing device fails 

Step 5: Measure ohms from pin to pin on one of suspect relays called 
out by self- test. 
Result: open 

Step 6: Measure ohms from pin to pin on another suspect relay. 
Result: 0 ohms 

Step 7: Measure ohms from pin to pin on another suspect relay. 
Result: 0 ohms 

Step 8: Replace relay tested in Step 5; rerun self-test 
Result: same fail 



The specific procedure displayed by the Experimental subject in this 
example is to investigate the control voltages to a relay before swapping it. 
The global principle is to thoroughly test all. possible influences on a 
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suspect component before swapping it. With the Control subject, we see that 
he was able to identify the faulty relay by taking measurements, not by 
swapping, as most Untutored subjects did. However, he failed to measure the 
control voltages before swapping the suspect component. A lack of 
understanding of the global principle that eschews premature swapping appears 
to be weak or nonexistent with the Untutored airman. 

CONCLUSION 

What we have reported here are results from a culminating study in a 
body of empirical work that has spanned ten years. The instructional approach 
calls for detailed cognitive task analysis results to enable a learning 
environment where students construct understanding in authentic contexts, 
rooted in the needs of practice. Results from other studies of the 
acquisition of complex, practical skills (such as electronic troubleshooting) 
have demonstrated how cognitive theories of learning and performance can 
enable improved apprenticeship training. Improvements seem attributable to 
two general instructional advances, which roughly correspond to better content 
and better method. First, cognitive theoretical models provide detailed 
representations of expert task performance to use as the targets of 
instruction . In the details, the goals to which procedural knowledge applies 
and the strategic processes that are responsible for the organization, 
coherence, and general execution of the performance are clearly established. 
With detailed cognitive models as input to instructional systems such as 
Sherlock 1 and 2, knowledge is directly tied to its uses in the world, and 
tacit knowledge (including goals, strategies, and assumptions) is made 
explicit for teaching. Content is thereby richer, more precise, and 
surrounded by context that establishes conditions of use. 

Second, better method has been achieved through a union of modern 
formulations of skill acquisition and traditional apprenticeship training 
techniques, such as modeling and coaching (Palincsar & Brown, 1984; 
Scardamalia & Bereiter, 1985; Schoenfeld, 1985). The common element in both 
is the notion of skill development as successive stages of increasingly mature 
performance. Hallmarks of apprenticeship training methods that are consistent 
with constructivist views include situated learning, external support or 
scaffolding in the instruction in the form of ideal modeling of the 
performance, hints, reminders, explanations, or even missing pieces of 
knowledge to assist the construction of understanding, and carefully sequenced 
learning activities that are both sensitive to changing student needs at 
different stages of skill acquisition and robust and diverse enough to foster 
integration and generalization of knowledge and skill (Collins, Brown, & 
Newman, 1987). Finally, to synthesize and reinforce the problem solving 
process, the solution steps are reflected upon , i .e. , inspected, evaluated, and 
compared to examples of more advanced solutions at the end of each session. 
The Sherlock tutors were designed with these "better methods" driving the 
instructional blueprint. 
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APPENDIX I: The F-15 Manual Avionics Test Station 



A test station is a large switch, more or less like a telephone 
exchange. It also contains instruments for measuring electrical energy 
patterns, such as a digital multimeter and an oscilloscope, and devices for 
creating patterned energy inputs to the aircraft component being tested. Each 
test on a box from an aircraft (called a line-replaceable unit (LRU) or 
sometimes the unit under test (UUT)) involves applying patterned electrical 
energy to various inputs of the UUT and then connecting various of its outputs 
to a measurement device. A central section of the test station, called the 
relay assembly group (RAG) mediates the switching process. The technician 
sets various switches on the front of the test station to soecify a particular 
test configuration, and then the RAG effects that configuration by energizing 
relays in giant switching trees. When all the relays are set properly, a 
signal circuit is created in which electrical inputs go from power supplies 
and signal generators on the test station, through an active connacting cable 
array (called the test package, TP), to the UUT, and outputs go from certain 
pins on the UUT's electrical interface through the switching array to a 
measurement device. On some test stations, a computer executes a series of 
tests of the UUT by directly controlling switching relays, but on the F-15 
manual station, switching is effected via control settings on the test 
station's front pan«:l . 

When a test station fails, this failure is manifest in some function 
that the test station does not perform properly. A first requirement in the 
face of a possible failure is to be sure that the abnormal outcome is not due 
to a fault in the unit under test, the box from the aircraft. This is the 
most likely situation - after all the whole purpose of the test station is to 
reveal faults in aircraft components. Another possibility that must be ruled 
out is a failure of the test package, the component that connects the UUT to 
the test station. If both the UUT and the Test Package are operating 
normally, then the problem is in the test station itself. 

The top-level diagnostic strategy would first attempt to isolate the 
problem into one of two main functional areas. Either the patterned energy 
inputs are not getting to the UUT, or its outputs are not getting to a 
measurement device successfully. A single test of the inputs of the test 
station to the UUT will reveal if the- problem is on the signal input side or 
the output measurement side. The next step is to trace the signal through the 
pathway, ending with an identification of a component that receives good 
inputs but has faulty outputs. If thi<^ component is involved in the switch;.ng 
process, there are two ways it could be failing. Either it is broken itself, 
or it is receiving wrong control signals fi'om the switches on the front panel 
of the test station. In this latter case, the control inputs tc the component 
in question will be wrong, and attention should be turned to diagnosis of the 
path from control switches to the component now being addressed. On the other 
hand, if a component ha<; good signal inputs, bad outputs, and good control 
inputs, then that component is a candidate for replacement. 
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This global strategy is really a combination of a weak or general method 
of space splitting, or "divide and conquer", with a specific model of the test 
station that provides an understanding of the meaningful units of the system 
that should be the focus of space splitting. The tactics for testing a 
particular subset of the test station, on the other hand, represent specific 
knowledge that can be generalized after appropriate experience and perhaps 
some expert suggestions. 
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